HDMQ: Towards In-Order and Exactly-Once Delivery using Hierarchical Distributed Message Queues

In today’s world distributed message queues is used in many systems and play different roles such as content delivery, notification system and message delivery tools. It is important for the queue services to be able to deliver messages in larger scales, at the same time it must be highly scalable and provide parallel access at the same time. An example of a commercial state of the art distributed message queue is Amazon Simple Queuing Service (SQS). SQS is a distributed message delivery fabric that is highly scalable. It can queue unlimited number of short messages (maximum size: 256 KB) and deliver them to multiple users in parallel. In order to be able to provide such high throughput at large scales, SQS confines some of the features that are provided by traditional queues. SQS does not guarantee the order of the messages. Furthermore, it also does not guarantee the exactly once delivery as duplicate messages can be delivered. This paper addresses these limitations through the design and implementation of HDMQ, a hierarchical distributed message queue. HDMQ consist of collection of area message nodes that can be used to store messages up to 512 KB. It utilized round robin local load balancer to save the message and scale across the area region accordingly. HDMQ provides 1 replica for high reliability of messages. HDMQ provides SQS-like APIs in order to provide compatibility with current systems that currently use SQS. We performed a detailed performance evaluation and compared HDMQ to SQS measuring throughput, latency and price per request. We found HDMQ to outperform SQS by up to 10-20% in throughput, 100% in latency, and 50% less in costs.